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ABSTRACT 


Software project development has been plagued with an infamous 
reputation for cost overruns, late deliveries, poor reliability and 
users dissatisfaction. Much of this blame has been placed on the 
manner in which software development projects are managed. The 
System Dynamics Model of Software Project Management is a 
quantitative model of software project dynamics that is attempting 
to gain some valuable insight into the managerial side of 
developing software systems. 

The objective of this thesis is to use the System Dynamics 
Model's gaming interface to investigate the effects of feedback on 
software project managers. Specifically, subjects were provided 
with either feedforward, outcome feedback, or cognitive feedback to 
determine which feedback form, if any, improved the subjects™ 
performance when confronted with a complex dynamic task, such as 
software project management. The results show that subjects in the 
cognitive feedback condition achieve a higher level of performance 
than those in either the feedforward or outcome feedback 


conditions. 


va Ha 


1b 


TABLE OF CONTENTS 


INTRODUCTION 

A. BACKGROUND 

B. EXPERIMENTATION TOOL 
C. RESEARCH QUESTION 


D. CONTRIBUTION 


THEORETICAL PREMISE 

A. FEEDBACK IN A DYNAMIC DECISION ENVIRONMENT 
1. Static vs Dynamic Decision Environments 
Zz. Inadequacy of Outcome Feedback 

B. ALTERNATIVES TO OUTCOME FEEDBACK 
1. Cognitive Feedback 
2. Feedforward 
3. Cognitive Feedback vs Feedforward 


C. HYPOTHESES 


METHOD 
A. TASK ENVIRONMENT 
B. MODEL 
C. EXPERIMENTAL DESIGN 
1. Between-Subjects 


2. Within-Subjects 


iv 


Oo j@Od oOo vn 


10 


a yt 


is 


Ls 


16 


18 


ee) 


IES, 


SUBJECTS 

i. Participant Profiles 
INFORMATION PROVIDED TO SUBJECTS 
1. Outcome Feedback 

2. Feedforward 

3. Cognitive Feedback 
EXPERIMENTAL SETTING 

DEPENDENT MEASURES 


EXPERIMENTAL RESULTS 


CONCLUSIONS 


SUMMARY OF RESULTS 


FEEDBACK AS A DECISION TOOL 


LIMITING FACTORS TO GENERALIZABILITY 


FUTURE RESEARCH 


APPENDIX 


LIST OF REFERENCES 


INITIAL DISTRIBUTION LIST 


20 


2 


ZZ 


Sage 


Za 


26 


32 


a3 


34 


CBS) 


oe, 


40 


41 


42 


43 


2s, 


62 


TABLE 


TABLE 


TABLE 


TABLE 


TABLE 


TABLE 


TABLE 


TABLE 


TABLE 


TABLE 


TABLE 


OR 


ee: 


LIST OF TABLES 


PROJECT CHARACTERISTICS 
EXPERIMENTAL DESIGN 
GROUP ASSIGNMENTS 
PARTICIPANT PROFILES. (Means ) 
PROJECT STATUS REPORT 
COGNITIVE FEEDBACK REPORT 


DEVIATIONS FROM INITIAL ESTIMATES. 
Means and (Standard Deviations) 


ANALYSIS OF VARIANCE. 
Deviation from Initial Estimates 


COMPARISON BETWEEN EXPERIMENTAL 
GROUPS AN BASELINE. Dependent Variable: 
Deviation from Initial Estimates : 


STAFF PRODUCTIVITY. Means and 


(Standard Deviations) 


ANALYSIS OF VARIANCE. 
Staff Productivity 


Vi 


Dependent Variable: 


Dependent Variable: 


}> 
18 
Zou 
Ze 
26 


Za 


32 


36 


on 


38 


a6 


Figure 
Figure 
Figure 
Figure 
Figure 
Figure 


Figure 


LIST OF FIGURES 


Model Structure 

Human Resource Management Subsystem 
WCWE Curve 

PrOTece. size Plot 

Project statfing and” sehedule Ploc 
Workforce Mix Plot 


Workforce Productivity Plot 


Vii 


I 


24 


a 


Zo 


Zo 


Si 


a2 





i INTRODUCTION 


A. BACKGROUND 

The proliferation of computing equipment over the past 
years has served to increase the demand for more reliable and 
complex software. Unfortunately, the success that has been 
common to the hardware industry has not been shared by those 
in the software industry. Today s software projects are 
typically delivered late and over budget. These inaccuracies 
have been blamed, in part, on ineffective software project 
managers. (Schlender, 1989) 

A large portion of these inaccuracies associated with the 
general project management problem can be attributed to the 
Geta cubty of control. One basic element, evident in any 
control system, is a means of transmitting feedback 
information to the control device (Anthony and Dearden, 1980, 
pp. 3-4). Control relies heavily on information feedback; the 
question is, however, what kind of feedback? 

There has been a great deal of research analyzing the 
effects of outcome feedback on management control, but this 
type of feedback has not been effective in improving the 
performance of decision makers. Research in static situations 
shows cognitive feedback to be more effective than outcome 


feedback in enhancing decision quality. However, little work 


has been done to determine how cognitive feedback may assist 
management control in complex dynamic tasks. 

Researchers have also suggested that the performance of a 
decision maker in a dynamic decision task, such as software 
project management, would improve if the decision maker’s 
model matches that of the task. Therefore, providing 
individuals with feedforward on a task may improve decision 
quality as opposed to outcome feedback. The focus of this 


thesis is on studying the effects of outcome feedback, 


cognitive feedback, and feedforward on one particular 
management control problem, that of software project 
management. 


B. EXPERIMENTATION TOOL 

The System Dynamics Model of Software Project Management 
(SDM) is a comprehensive model of the software development 
process that integrates both the managerial and software 
development activities. Through the use of a model, 


The effects of different assumptions and environmental 
factors can be tested. In the model system, unlike the 
real systems, the effect of changing one factor can be 
observed while all other factors are held unchanged. Such 
experimentation will yield new insights into’ the 
characteristics of the system that the model represents. 
By using a model of a complex system, more can be learned 
about internal interactions than would ever be possible 
through manipulation of the real system. Internally, the 
model provides complete control oof the system's 
organizational structure, its policies, and its 
sensitivities to various events. (Forrester, 1961, p.1) 


Additionally, this particular model provides an effective 
means of studying dynamic decisions. 

The gaming interface of the System Dynamics Model provides 
experimenters with the ability to analyze the efforts of any 
number of software project managers. The experimenter can 
vary the type of feedback given to the manager by specifically 
tailoring the model’s interface for that particular manager. 
The model provides the capability of displaying a wide variety 
of variables in either tabular or graphical form. The results 
of each manager's run can then be collected and analyzed to 
determine any particular trends in their decision making 


process. 


C. RESEARCH QUESTION 
Recent laboratory experiments have provided valuable 
insight into human behavior ina variety of decision-theoretic 
contexts. This research, however, has focused mainly on 
static and discrete judgements. As Hogarth (1981) emphasizes 
...the continuous, adaptive nature of the judgmental 
processes used to cope with a complex, changing 
environment.... With few exceptions...judgment 
researchers have focused on discrete incidents (particular 
actions, predictions, and choices) that punctuate these 
continuous processes; furthermore, task environments are 
typically conceptualized to be stable. (p. 198) 
Sterman (1989a) argues that experimental studies of the 
"continuous, adaptive nature of judgmental processes" in a 


dynamic system, such as software project management, can be 


conducted in the laboratory with the aid of computer 


Simulation models. He adds that "simulations can represent 
the structure and complexity of such systems with great 
fidelity and permit controlled manipulations of the decision 
context and information presented to the subject." 

As an example, the research question addressed in this 
thesis is: What effects do cognitive feedback, outcome 
feedback, and feedforward have on decision makers ina dynamic 


decision environment such as software project management? 


D. CONTRIBUTION 

Enhancement of management control through the use of 
cognitive feedback has attracted much attention (Kleinmuntz, 
1985; Sterman, 1989a). The use of cognitive feedback to aid 
software project managers has not, however, been investigated. 
The goal of this research, therefore, is to establish the 
importance of cognitive feedback as an aid to the decision 


making process of software project managers. 


Tl. THEORETICAL PREMISE 


A. FEEDBACK IN A DYNAMIC DECISION ENVIRONMENT 
1. Static vs Dynamic Decision Environments 
When analyzing human judgmental ability, it is 
important to distinguish between static and dynamic decision 
environments. Although much of human decision making is 
composed Ot discrete incidents (Particular actions, 
predictions, and choices) occurring in a seemingly static 
environment, these incidents are only a subset of, and serve 
to punctuate, our continuous processes which occur in response 
to our dynamic environment. As Hogarth (1981) indicates, the 
limitation of existing research on human judgment is that it 
focuses only on these discrete incidents in - static 
environments. Since most decisions are made in a continuous, 
dynamic environment, it is argued that biases observed during 
these discrete incidents occur as a result of heuristics that 
are derived from man’s more natural continuous environment. 
According to Hogarth (1981), failure to evaluate human 
judgement as a continuous process has two distinct pitfalls: 
First, insufficient attention has been paid to the effects 
of feedback between organism and environment. Second, 
although judgmental performance has been evaluated 
according to the principles of optimal behavior implied by 
decision theory and the probability calculus, few 


researchers have questioned whether the assumptions of 
such models apply to continuous processes. (p. 198) 


Studies involving human decision making cannot 
overlook the importance of feedback. Most all human judgement 
is used to facilitate an action and this action is most often 
followed by immediate feedback. Our next action is then 
directly influenced by this feedback causing a action, 
outcome, feedback, action loop. Hogarth (1981) indicates that 
the tendency to overlook feedback as a crucial part of this 
loop comes as a result of its “ubiquity"™ in the environment. 
As Powers states (1973): 

All behavior involves strong feedback effects, whether one 
is considering spinal reflexes or self-actualization. 
Feedback is such an all-pervasive and fundamental aspect 
of behavior that it is as invisible as the air we breathe. 
Quite literally it is behavior--we know nothing of our own 
behavior but the feedback effects of our own outputs. (p. 
Sis wile) 
2. Inadequacy of Outcome Feedback 

A majority of the work that has been done in relation 
to the dynamic decision environment has examined the effects 
of outcome feedback on the decision making process (Brehmer, 
1987; Sterman, 1989b). Evidence, however, indicates that 
presenting outcome feedback in a dynamic environment has 
dysfunctional effects that persist over time. These effects 
fall into four categories. 

First, subjects typically misperceive time lags in the 
system which confronts’ them. In Sterman’s (1989b) stock 


management problem, subjects fail to adequately account for 


the supply line. Subjects confronted with Brehmer’s (1987) 


DESSY experiment show improvement after spending two hours a 


day for four days with the system, "but only if there are no 


delays in the system. If there are even minimal delays, the 
subjects control over the system does not improve." As 
Brehmer states, "this [fact] is somewhat disconcerting, since 


delays are probably a more common case than that of immediate 
feedback." 

The second dysfunctional effect that typically plagues 
subjects in outcome feedback experiments 18S a wide oscillation 
of results over time (Sterman, 1989b). In Sterman’s stock 
management problem, this oscillation is seen in the inventory 
level. Subjects in Sterman’s experiment also attribute the 
dynamics of the system to external variables rather than as a 
direct result of their interactions with the environment. 
Thus, subjects misperceive the feedback from their own 
decisions. The final dysfunctional effect of outcome feedback 
is seen in (Wagenaar, 1985) where subjects misperceived 
exponential growth over time (and hence, nonlinear changes). 

Experimental evidence indicates that outcome feedback 
is not an adequate aid in decision making. As Sterman (1989b) 
states: "The results here suggest that outcome feedback alone 
is not sufficient: by attributing the source of change to 
external factors, people s mental models lead them away from 
the true source of difficulty." Kleinmuntz and Thomas (1987) 
drew similar conclusions: "Despite the corrective benefits of 


outcome feedback..., it may still be quite difficult to learn 


how to improve one’s decision rules using outcome feedback 
alone." The inadequacy of outcome feedback in dynamic 
environments has led researchers to explore alternative means 


of improving performance. 


B. ALTERNATIVES TO OUTCOME FEEDBACK 
1. Cognitive Feedback 

In contrast to outcome feedback, which provide - 
subjects with information about the accuracy or correctness of 
their response, cognitive feedback represents "information 
regarding the how or why that underlies this accuracy." 
(Jacoby et al., 1984) Doherty and Balzer (1988) describe 
cognitive feedback as information which provides subjects with 


the following relationships: 


1. Between cues and criterion, i.e., information about the 
task. This is known as task information and it is 
characterized by three kinds of relational indices-- 
overall task predictability (Re), cue intercorrelations 
(rij), and correlations between cues and the criterion 
(rie). 


2. Between cues and the persons inferences, 1. 
information about the person’s cognitive state. This is 
known as cognitive information and, in terms of the lens 
model, largely mirrors task information (the only 
exception being cue intercorrelations which is not 
represented in this relationship). 


3. Between cognitions and the distal objects. This third 
category comprises indices of "functional validity” 
information, or, information about the relation of the 
cognitive system to the task system (Balzer et al., 
1989). These indices include the achievement correlation 
(ra), the matching index (G), and the correlation between 
the residuals from the predictions of those models (C). 


Since Cognitive Feedback has been used largely for 
static tasks in the context of the lens model, the aim of this 
study is to extend this notion to a dynamic decision situation 
using a system dynamics model. As Sterman’s (1989b) research 
indicates "the efficacy and robustness of decision strategies 
lies not only in the availability of outcome feedback, but 
depends crucially on the nature of the action feedback between 
decisions and changes in the environment which condition 
future decisions." Brehmer (1987) concludes from his DESSY 
experiment that, "results on verbalization suggest that 
information about the system may need to be communicated in 
nonverbal form, and that various graphic displays may prove 
useful." Additionally, Kleinmuntz and Thomas (1987) state 
that "feedback about the decision process being used may also 
be valuable...." Each of these statements support the belief 
that cognitive feedback should prove more beneficial than 
outcome feedback when presented to subjects confronted with a 
dynamic decision situation. 

2. Feedforward 

Another method of assisting subjects in forming mental 
models of complex systems is through feedforward. Feedforward 
can be defined as "the transmission of task information 
directly to the subject." (Bjorkman, 1972) Studies by Conant 
and Ashby (1970) showed that in order to perform effectively 


with a dynamic process, an operator must have a model of the 


system. Bjorkman (1972) provides three hypotheses with regard 
to the use of feedforward in Knowledge and policy formation: 

1. Knowledge acquired by feedforward should be more accurate 
and consistent since it does not suffer from various 
sources of error and bias due to the trial--by--trial 
accumulation of information. 

2. Feedforward relieves the learner from a certain amount of 
cognitive strain since he already Knows things which 
otherwise should have been learned by feedback. This may 
give the learner an increased opportunity to focus on 
policy formation. 

3. It seems reasonable to assume that feedfor rd favors an 
analytical rather than intuitive mode of thought. 
Uncertainty, which is one of the factors that contributes 
to intuitive rather than logical, stepwise inference, has 
been removed entirely or partly by feedforward. 

The operationalization of feedforward occurred, for the 
purposes of this study, through the use of a special training 
session which assisted subjects in forming a model of the 
software project management system. 
3. Cognitive Feedback vs Feedforward 

Although feedforward assists subjects in forming a 
mental model of the system, task information is presented only 
prior to performing the task (or, at best, the information is 
presented prior to performing the task and the same 
information is available to the subject throughout the task). 
As shown by Morris and Rouse (1986), "a priori knowledge can 
be a powerful basis for gaining new knowledge or, if 


incorrect, an impediment to gaining correct knowledge." A 


distinct advantage gained through cognitive feedback is that 
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subjects are constantly being presented with task information 
so that they may change their mental models of the system to 


meet changes experienced in the environment. 


C. HYPOTHESES 

Two primary hypotheses guide the research question. The 
first hypothesis 1S concerned with the relationship between 
feedback condition and subject performance. As indicated in 
sections 1 and 2, cognitive feedback and feedforward assist 
subjects in forming a mental model of the task. Thus, 
subjects receiving this information should perform better than 
those not receiving this information. Additionally, cognitive 
feedback continually assists subjects in keeping their mental 
model current with the dynamic environment. One would 
therefore expect subjects receiving cognitive feedback to also 
outperform those subjects receiving feedforward. This leads 
us to the first hypothesis: 

Subjects receiving cognitive feedback will perform better 
than subjects receiving either outcome feedback or 
feedforward. 

The second primary hypothesis addresses the characteristic 
of the task which confronts the subject. This is accomplished 
by measuring each subjects performance during each of three 
software projects with varying degrees of complexity (details 
on the three projects, referred to as ideal, fixedsize/bad 
estimates, and undersize will be provided in Chapter III). We 


would expect that subjects would perform better when 


ala 


confronted with a task with less complexity. Since the ideal 
project has fewer lags and less of the "noise" associated with 
more complex tasks, it is regarded as the least complex of the 
three projects. The second hypothesis is therefore: 
Subjects performance while managing the ideal software 


project will be better than when managing either the 
fixedsize/bad estimates or undersize software projects. 


WA 


III. METHOD 


A. TASK ENVIRONMENT 

The task that subjects were asked to perform was in many 
ways similar to flight simulators that pilots use to mimic 
flying an aircraft from takeoff at point A to landing at point 
B. Instead of flying an aircraft, though, the simulation 
mimicked the life of three real software projects from the 
start of the design phase until the end of testing. Subjects 
were more than outside observers, however, they performed an 
actual role in the project: that of the project manager. 

Specifically, subjects were required to track each 
project’ s progress using a number of reports generated by the 
project team at different intervals throughout the project 
life. They then made project staffing decisions based on the 
knowledge gained from those reports. As project manager, 
subjects were permitted to hire additional staff or decrease 
the staffing level as deemed necessary to complete the 
project. Their objective in setting the staffing level was to 
decide on the best compromise between finishing on an 
acceptable schedule while avoiding an excessive cost overrun. 


Specifically, subjects attempted to: 


1. complete the project on schedule, 


Z. at the lowest possible cost, and 
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3. in any case, complete it before the maximum tolerable 
completion date. 

Unlike Sterman s (1989b) use of a Generic Stock-Management 
System and Brehmer’s (1987) DESSY experiments which presented 
subjects with open-ended tasks, the task which confronted each 
subject in our experiment was close-ended with each project 
having a finite completion point. 

The task of managing a Software Project was selected for 
several reasons. First, Software Project Management has all 
of the characteristics of a dynamic problem (Brehmer and 


Allard, 1985): 


1. It requires a series of decisions. 


2. The environment changes both spontaneously (staff 
productivity, changing requirements, etc) and, as a 
consequence of the decision maker’ s actions. 


3. The time element is critical; it is not enough to make 
the correct decisions and to make them in the correct 
order, they also have to be made at the correct moment in 
time. 


Like Brehmer’s (1987) DESSY experiment, the Software Project 
Management problem is interesting 


...because the standard normative theories for decision 
making do not apply (Brehmer and Allard, 1985); the models 
of the task embodied in these theories simply do not fit 
this kind of task. It is not possible to compute the 
correct course of action. This can only be found from a 
model of the system and, before the operators have 
developed such models, they will not be able to control 
the system. The research problem, is whether or not 
people are able to develop good mental models of this and 
Similar tasks. (p. 24) 
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Finally, Software Project Management is currently a critical 
issue due to the frequency of projects that are delivered over 
budget and late (Schlender, 1989). 

Three separate projects, referred to as ideal, 
fixedsize/bad estimates, and undersize, were selected in an 
attempt to cover the spectrum of projects that typically 
confront Software Project Managers. Table 1 shows’ the 


characteristics associated with each of the three projects. 


TABLE 1. PROJECT CHARACTERISTICS 


Ideal Undersize Fixedsize 

Project cost: initial estimate 3,721 1,460 2,972 
(Man-days ) 

Project Size: initial estimate 1,067 397 1,866 
(No. of tasks) 

Actual Size of Project 1,067 610 1,866 
(No. of tasks) 

Project duration: initial estimate 413 362 380 
(Days ) 

Maximum tolerable project duration 479 420 441 
(Days) 


Notes: 1. The ideal project had accurate initial estimates of project size and cost. 
2. The undersize project had understated initial estimates of size, and therefore, 
cost. 


3. The fixedsize project had an accurate initial estimate of size. The initial cost 
estimate was, however, understated. 


Subjects were presented with accurate initial estimates as 
well as accurate information throughout the entire lifecycle 
of the ideal project. Subjects were given accurate initial 
estimates for the fixedsize/bad estimate project, however, 
estimates given during the project lifecycle were typically 
unreliable. The undersize project was a project that grew in 


size from an initial estimate of 397 tasks to 610 tasks at 


i> 


project completion. This growth in project size was 


attributable to changing user requirements. 


B. MODEL 

The Model of Software Project Management attempts to 
provide "a comprehensive model of the dynamics of software 
development that enhances our understanding of, provides 
insight into, and makes predictions about the process by which 
software development is managed." (Abdel-Hamid and Madnick, 
1989) Figure 1 shows the model with its four subsystems: the 
human resource management subsystem, the software production 
subsystem, the controlling subsystem, and the planning 
subsystem. "The model was developed on the basis of a battery 
of 27 field interviews of software project managers in five 
software producing organizations, supplemented by an extensive 
database of empirical findings from literature." (Abdel-Hamid, 
1989) The human resource management subsystem accounts for 
variables related to the workforce, namely, the hirizg rate, 
training, and turnover of project personnel. The software 
production subsystem models the designing, coding, and testing 
phases of the software development lifecycle. This subsystem 
also accounts for the quality assurance effort required for 
project develop as well as the actual productivity of the 
project team. In contrast to actual productivity, perceived 
productivity is described in the control subsystem. Perceived 


productivity directly influences a manager s estimate of 
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Model Structure 


Figure 1. 
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project tasks perceived to be completed. This estimate, 
however, is often unrealistic with regard to software 
development since one must have accurate knowledge of rates of 
accomplishment and resources expended to date (Abdel-Hamid and 
Madnick, 1989). Thus, this variable often is no more than a 
measurement of budgeted resources that have been expended. 
The planning subsystem, the final subsystem of the model, 
provides initial project estimates such as project cost, 
schedule, and staffing. As the project continues through its 
lifecycle, these estimates will change to reflect management ’s 


decisions. 


C. EXPERIMENTAL DESIGN 

The research design is illustrated in Table 2. The 
experiment used a factorial design with two components in 
order to capture the feedback condition and the project type. 


These components are between-subjects and within-subjects. 


TABLE 2. EXPERIMENTAL DESIGN 


Type of information 


Cognitive Feedback Outcome Feedback Feedforward 
Project No. Project No. Project No. 
2 3 1 2 3 1 2 3 
Order 
Order 1 I U F I U F I U F 
Order 2 U I F U I F U I F 
Order 3 F U I F U I F U I 


Notes: 1. Participants were randomly assigned to one of the feedback conditions and one of 
the sequences of task conditions. 
2. I, Uy and F refer to ideal, undersize, and fixedsize projects, respectively. 
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1. Between-Subjects 
The fundamental objective driving the experiment is to 
determine the effect of cognitive feedback, i.e., determine 
how best the operator can be conveyed a model of the system 
over time (Brehmer, 1987): through feedforward, outcome 
feedback, or cognitive feedback. This is the rationale for 
the between-subject component. 
2. Within-Subjects 
In addition to determining if systematic differences 
exist among experimental conditions, feedback studies also 
seek to study the effect within each condition over time. 
This is referred to as within-subject design (Barlow and 
Hersen, 1984, p. 66) and involves multiple measurements over 
different points in time. In this experiment, the within 
subjects aspect was operationalized by using three separate 
projects, namely, the ideal, fixedsize/bad estimate, and 
undersize projects. 
Randomization between and within subjects was achieved 
using a Latin Square Design as follows (Kirk, 1982, pp. 3ll- 
SZ): 
First, each project was assigned a corresponding letter. 
A: Ideal Project(IL) 
B: Fixedsize/bad estimate Project(FB) 
C: Undersize Project(UN) 
Next, two sequences of random numbers were generated. 


(2.515 3:). 43.402.) 


1g 


The appropriate square is then selected. 


A B C 
B ce A 
C A B 


Rows are then ordered according to the first set of random 


numbers. 


Q YW 


cS A 
B Cc 
A B 
Next, columns are ordered according to the second set of 


random numbers. 


A B C 
C A B 
B C A 


Finally, the notation is converted according to project name. 


Gl G2 G3 
IL FB UN 
Task UN IL FB 
Order FB UN IL 


Therefore, Group 1 receives the projects in the order: ideal, 


undersize, and fixedsize/bad estimate. 


D. SUBJECTS 
The experiment was conducted using 56 graduate students. 
Participants were divided into nine groups based on the 
feedback condition and task order. Table 3 shows the 
feedback condition and task order provided to each group. 
Each subject was assigned a number from 1 to 56 according 
to the alphabetical order of his last name. Two digit random 


numbers were then generated using a random number table. 
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TABLE 3. GROUP ASSIGNMENTS 
GROUP NUMBER FEEDBACK CONDITION/TASK ORDER 


Cognitive Feedback, Gl Task Order 
Cognitive Feedback, G2 Task Order 
Cognitive Feedback, G3 Task Order 
Outcome Feedback, Gl Task Order 
Outcome Feedback, G2 Task Order 
Outcome Feedback, G3 Task Order 
Feedforward, Gl Task Order 
Feedforward, G2 Task Order 
Feedforward, G3 Task Order 


WEN OWRD PD RO met et et 
! 
WER = WER = WER 


Subjects corresponding to the first six numbers were assigned 
to group 1-1, the next six in group 1-2, etc. Duplicates and 
random numbers greater than 56 were disregarded. Due to the 
number of subjects not being evenly divisible by nine, groups 
1-3 and 2-3 had seven subjects each. 
1. Participant Profiles 

Two types of subject characteristics could potentially 
have affected results in this experiment: demographic factors 
and task-specific factors. Demographic factors were 
operationalized as age, full time work experience, years since 
completion of undergraduate education, familiarity in working 
with computers and hours per week a subject spent working on 
computers. Table 4 profiles the subjects with respect to 
demographic factors. The task-specific factor was 
operationalized by asking subjects whether they had any prior 
experience in the task. It was determined that none of the 
subjects had any significant experience in software project 


management. 
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TABLE 4. PARTICIPANT PROFILES. (Means). 





By feedback condition: 








Cognitive Outcome 

Feedback Feedback Feedforward 
AGE 34 33 31 
WK_EXP 14 13 10 
Y_UGRAD 10 10 8 
F AM_COMP 5 6 5 
HRS_COMP 10 9 9 
By project order: 

Order 1 Order 2 Order 3 
AGE 34 30 34 
WK_EXP 14 10 13 
Y_UGRAD 10 8 1] 
FAM_COMP 6 5 5 
HRS_COMP 12 8 9 


Key: AGE = Age of subjects (Years) 
WK_EXP = Full time work experience (Years) 
Y_UGRAD = Years since completion of undergraduate education 
FAM_COMP = Familiarity of subjects with computers (1 = not familiar, 9 = very familiar) 
HRS COMP = Hours per week spent using computers 
E. INFORMATION PROVIDED TO SUBJECTS 


Subjects were provided different types of information 
based upon the feedback condition corresponding to their 
group. During the lifecycle of each project, the experiment 
software would pause at 40 day intervals to allow subjects to 
review this information. 

1. Outcome Feedback 

Subjects receiving outcome feedback were given only 
one report, the Project Status Report, at the end of each 
interval. Information provided in this report is shown in 


Table 5. 


TABLE 5. PROJECT STATUS REPORT 





CURRENT INTERVAL STATISTICS: Elapsed Time = 80 Days 
INITIAL ESTIMATES: (These will not change throughout the project) 

Project size 1,067 Tasks 
Project duration 413 Days 
REPORTED STATISTICS at Time = = = = => 80 Days 

Z Development Reported to be complete 10.26 Percent 
% Testing Reported to be complete 0.00 Percent 
Perceived Total Project Size at this point 1,066.67 Tasks 
Perceived Total Project Cost at this point 3,721.00 Man Days 
Total Number - Fulltime Equivalent Staff G.3 Fulltime staff 
New Estimate of Project Duration (start - end) 829 Days 
Maximum Tolerable Completion Date 479 Days 
Total Man Days Expended 348.36 Man Days 
Total Number of Tasks developed to date 137.45 Tasks 


PRESS <ENTER> TO RETURN TO MENU 


2. Feedforward 

In addition to receiving the Project Status Report 
(Table 5), subjects in the feedforward groups were given a 
separate training lecture prior to the experiment which 
provided further insight into the human resource management 
subsystem of the Model of Software Project Management. Figure 
2is an exploded view of this subsystem. 

The first part of the feedforward training provided 
SUD jects With instruction on two concepts ¢ritical -in the 
human resource management subsystem: average productivity and 
net cumulative contribution. To demonstrate the importance of 
carefully considering each of the two concepts, the following 
human resource management problem was given: 

The initial project team consists of five people each 
with a productivity of ten lines of code (LOC) per man day 
(MD) thus giving a total output of 50 LOC per day for the 


entire team. The assumption is that the project is behind 
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Figure 2. Human Resource Management Subsystem 


schedule and the manager must make a decision either to add a 
new person to the team or accept the schedule slippage. 

Case one examines the effect of adding a new person 
with a productivity of 8 LOC per man day. If this person is 
added, it is expected that the productivity of the old team 
will decrease by 10 percent (i.e., 9 LOC/MD) due to training 
and the added communication overhead. Case two also adds a 


new person but this person’s productivity is only 4 LOC per 
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man day. Again, this will cause a 10 percent decrease in the 
productivity of the old team. 

In the first case the output of the team increases to 
53 LOC/Day (given by 5 team members x 9 LOC/MD + 8 LOC/MD for 
the new member). The average productivity of the team is now 
8.8 LOC (53 divided by 6 team members). The net cumulative 
contribution of the new person is 53-50 or 3 LOC/MD. Since 
the average productivity of the team decreases, the cost of 
the project will increase, but since the net cumulative 
contribution of the new person is positive, the schedule will 
go down. 

In case two, the output of the team is only 49 LOC/Day 
(5 team members x 9 LOC/MD + 4 LOC/MD for the new team 
member). The average productivity of the team is 8.1 LOC (49 
divided by 6 team members), and the net cumulative 
contribution of the new person is 49-50 or -1 LOC/MD! Thus 
the addition of the new team member is detrimental to the 
preeject, not only driving up the project s cost but its 
schedule as well. 

Although the mathematics of the concepts are 
relatively simple, the importance of the lesson is to realize 
that adding a person (or people) to a late project will not 
always improve the project’s schedule. The manager must look 
closely at the average productivity of the project team as 
well as the net cumulative contribution of any team members 


added. 
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management subsystem. 


The second part of the feedforward training lecture 
focused on considerations involved in the willingness to 
change workforce (WCWF). Subjects were presented with the 
equation, 


Workforce = Indicated * WCWF + Current * (1-WCWF) 
Sought Workforce Workforce 


and its relation to the WCWF curve (Figure ae The 
instructor explained to subjects that a manager at a point in 
the project which yields a WCWF value of zero from the curve 
is, in essence, not willing to change his workforce and thus, 
the workforce sought will be equal to the current workforce. 
If, however the manager is at a point in the project where the 
WCWF is one, the manager is very willing to change the 
workforce and thus, the workforce sought is equal to the 
indicated workforce.’ 
3. Cognitive Feedback 

Subjects receiving cognitive feedback also received 
the Project Status Report (Table 5) after each 40 day 
interval. Additionally, personnel receiving cognitive 
feedback had the option to view a Cognitive Feedback Report as 


well as four plots. The Cognitive Feedback Report (Table 6) 


} 


training time, set at 20 days. 


Subjects were told that the time parameter referred to in 
the figure was the sum of two parameters from SDM s human resource 
These two parameters are the hiring delay, 
set at 30 days for this experiment, and the assimilation or 


‘The indicated workforce is ynonymous w ‘h the workfc-ce 
necessary to stay on schedule. 
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Figure 3. WCWF Curve 


TABLE 6. COGNITIVE FEEDBACK REPORT 


CURRENT INTERVAL STATISTICS: Elapsed Time = 80 Days 

INITIAL ESTIMATES: (These will not change throughout the project) 

Project Size 1,067 Tasks 

Project Duration 413 Days 

REPORTED STATISTICS at Time = = = = = = => 80 Days 

Fraction of Workforce that is Experienced 0.9 

Perceived Average Productivity 0.4 Tasks/Man-day 

Communication Overhead 0.01 

Total Number - Fulltime Equivalent Staff 4.3 Fulltime Staff 

Estimated Workforce Needed to Stay on Schedule 4.5 Fulltime Staff 
PRESS <ENTER> TO RETURN TO MENU 

provided information on specific workforce variables. Four 


plots, described as the Project Size Plot, the Staffing and 


Schedule Plot, the Workforce Mix Plot, and the Workforce 


2/ 


Productivity Plot, were providéd to assist *subjéctS Wan 
spotting trends dev@loping throughout the project lite®cyelee 

They plot on Project Size (Figure 4) plotted tva 
variables, the perceived total project size to date (PJBSZ) 
and the perceived total project cost to date (JBSZMD). ovér 
thes project lifecycim. This plot provided subjects 
information on whether any schedule or budget slippage was a 
result of either unexpected increases in the project's size 
(e.g., due to changes in users requirements), or that the 


effort required to complete the project was initially 


Porepived tatnl proajact 
cost. to data | 


Parceived total projact 
siza to data 


i ue 


120 160 


Time (Days) 





Figure 4. Project Size) Plot 
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underestimated (e.g., because the project complexity was 
underestimated). If the former case were true, subjects 
would expect to see the variable PJBSZ increase over time. If 
the latter were true, then the variable JBSZMD would increase 
over time. 

The plot on Project Staffing and Schedule (Figure 5) 
plotted three variables of the project lifecycle: the total 
number of fulltime equivalent staff (FTEQWE), the new 


estimate of project duration from start to end (SCHCDT), and 


'law estimate of 
profact duratlon 


'Worktorce naeded 
tO Stay on : 
schadule 


Futitime 
ebuivatent 


| En 
120 160 ate 


reporting 


Time (Days) perlod 





Figure 5. Project Staffing and Schedule Plot 
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the estimated workforce needed to stay on schedule (WFINDC). 
This plot provided feedback on the trade-off between 
minimizing schedule over-runs versus minimizing cost over- 
runs. When a project runs into difficulties, a manager can 
choose to stick with the project's schedule (SCHCDT) by 
increasing the workforce level (FTEQWEF). This practice always 
increases the cost of a project. On the other hand, a manager 
might wish to minimize his/her cost over-run, by avoiding an 
increase in the workforce level, and instead, opt to increase 
the project’ s scheduled completion date. The indicated 
workforce level (WFINDC) is provided as an estimate of the 
workforce needed to stay on schedule. 

The plot on Workforce Mix (Figure 6) provided subjects 
with feedback on their staffing decisions. In general, 
staffing decisions have the greatest impact on productivity. 
This option plotted three variables: the total number of 
fulltime equivalent staff (FTEQWF), the fraction of the 
workforce that iS experienced (FRWFEX), and the planned 
workforce (PLANWE) over the project lifecycle. This plot was 
deemed useful for two reasons. First, larger workforces will, 


in most cases, be less productive because of the increases in 


communication and training overheads. Second, the workforce 
mix (i.e., percent of experienced vs new staff in the 
workforce) will also have an impact on productivity. The 


larger the percentage of experienced people in the workforce, 


the more productive the workforce. 
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Figure 6. Workforce Mix Plot 


The plot on Workforce Productivity (Figure 7) provided 
subjects with information on the average productivity of their 
team. It displayed the relationship between the total number 
of fulltime equivalent staff (FTEQWF), the perceived average 
productivity of the workforce (ASSPRD), and the communication 
overhead associated with the workforce (COMMOH) throughout the 
project lifecycle. The basis for presenting this plot was, 
again, twofold. First, the larger the workforce, the lower 
the average productivity. This is due to the time people 


"waste" in communicating with teammates. Second, the 


ae yall 


communication overhead curve shows subjects exactly what 


percentage of a person’s time is wasted in communication with 


others. 
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Figure 7. Workforce Productivity Plot 


FEF. EXPERIMENTAL SETTING 

The experiment was conducted in two computer labs with two 
attendants per lab. Each subject was assigned a specific 
terminal and was given documentation (see Appendix) and a disk 
according to his/her group assignment. Subjects were given 
time to read over the documentation and ask questions about 


the conduct of the experiment. Questions pertaining to the 
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task were not permitted. After reading the documentation, 
subjects ran several intervals of a "dummy" project to 
familiarize them with the reports, plots and keystrokes 
required to traverse through the screens they were presented 
Walt hy: Attendants assisted the subjects with any technical 
difficulties. After completing the trial run, subjects were 
permitted to proceed with each of the three projects at their 


own pace. 


G. DEPENDENT MEASURES 

Two dependent variables, deviation from initial estimates 
and staff productivity, were used to analyze the performance 
of each subject. Two numbers were used to determine the 
deviation from initial estimates. The first number was the 
difference between the subject's completion time and the 
estimated completion time. The second number was” the 
difference between the subject's final cost and the estimated 
project cost. These numbers were then averaged to yield the 
deviation (overrun or underrun) from the initial project 
estimates. Project time is defined as the length, measured in 
days, required for the subject to successfully manage each 
project from start to end. Project cost measures’ the 
resources expended, in man days, to complete each project. 
The staff productivity is defined as tasks per man-day and was 


determined by dividing the number of tasks associated with 
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each project by the total cost of that project 1en managed by 
a particular subject. 

Lower benchmarks for time and cost were determined for 
each of the three projects by running 15 simulations for each 
project using random staff sizes. Minimum and maximum staff 
Sizes for each project type were determined from subjects’ 
decisions. Five hundred random numbers were then generated 
for each project to fall within the minimum and maximum staff 
size. The 15 simulations were then run for each project using 
staff sizes taken sequentially from the 500 generated in that 


project’ s staff range. 


H. EXPERIMENTAL RESULTS 

Tables 7 through 11 summarize the results of the 
experiment. Cases in which subjects made significant errors 
were discarded, and data from 45 subjects was retained for 
analysis. Table 7 shows performance between subjects, as well 
as within subjects, with respect to deviations from the 
initial project estimates. 

Applying the approach suggested by Winer (1971, p. 697), 
the following analysis of variance (ANOVA) model suited for 
multiple Latin Squares was used to test Hypotheses: 

Yigkaym™ = HW + ay + By + Te + Xp + OQ + 
(TX), + (Fa), + (58 ),; + GijkC)n where: 
“ 1s constant, 


a. is the sequence of the projects (1 = 1,2,3), 


1 
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B, NS lene Ordervor Che projece (79. —= 1,2,3), 


By is the feedback condition (kK = 1,2,3, where l = 
cognitive feedback, 2 = feedforward, and 3 = outcome 
feedback), 


Myels ene. eype,of projece (i) =11,2,3, where 1 = ideal, 
2 = undersize, and 3 = fixedsize/bad estimate), 


6, 1s the experimental participant (m = 1,...,45), 


n 


Cijk(jn 2S the experimental error term. 


TABLE 7. DEVIATIONS FROM INITIAL ESTIMATES. Means and (Standard Deviations). 


N Ideal Undersize Fixedsize 
Cognitive Feedback 15 0.448 11.501 39.29 
(1.723) (4.43) (1.854) 
Feedforward 15 3.858 17.684 47.21 
(1.794) (2.058) (5.605) 
Outcome Feedback 15 10.019 25.753 57.027 
(5.710) (6.052 ) (7.041) 
Random Baseline 15 11.12 27.98 56.22 
(6.12) (7.23) (9.12) 


The analysis was conducted with the General Linear Models 
procedure (SAS, 1987). Table 8 contains the ANOVA results. 
The results show that the performance of subjects across the 
three different feedback conditions was’ significantly 
different (F=345.89, p=0.0001). The null hypothesis of no 
Significant differences among feedback conditions is therefore 
rejected. In other words, the results indicate that the 
subjects performance was influenced by the type of 


information provided to them. 
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TABLE 8. ANALYSIS OF VARIANCE. Dependent Variable: Deviation from Initial Estimates. 








Source of Degrees of 

Variance Sea. freedom F-Value P R-Square 
Model 50769.00 57 136.54 0..901 0.89 
Type of 

Information 4530.01 2 345.89 0.0001 

Type of Task 44166.85 2 3372.57 0.0001 

Sequence 16.53 2 1.26 0.2887 

Order 3.16 2 0.24 0.7862 
Participant 2041.19 4] 7.60 0.0001 

Group*Task 4.48 4 0.23 0.8126 
Group*Order 6.67 % 0.26 0.9036 

Within Groups 504.22 77 


Additionally, Table 8 shows that the performance within- 
subjects was significantly different depending on the type of 
project confronting them (F=3372.37, p=0.0001). The null 
hypothesis of no significant differences in performance 
depending on type of project presented is therefore also 
rejected. 

The Tukey Test for Additivity indicated no presence of 
interaction effects between the sequence of projects (p>0.2), 
or the order in which projects were completed (p>0.7). Also, 
there were no interaction effects between the type of 
information provided and the type of project (p>0.8), nor the 
type of information provided and the order of project (p>0.9). 

Table 9 summarizes tests comparing results from each of 
the feedback conditions with the random baseline with respect 
to deviations from initial estimates. The mean total 
deviation for subjects in the cognitive feedback and 


feedforward conditions was lower than the random baseline in 


TABLE 9. COMPARISON BETWEEN EXPERIMENTAL GROUPS AND BASELINE. Dependent Variable: 
Deviation from Initial Estimates. 





Ideal Undersize Fixedsize 
Comparison Total Total Total 

deviation deviation deviation 
Cognitive Feedback r>cfb r>cfb r>cfb 
Feedforward roff rert r>ft 
Outcome Feedback n.s. n.s. n.s. 


Notes: 1. cfb: cognitive feedback, ff: feedforward, n.s.: not significant, rt: random 
baseline. 
2. The comparisons represent one-tailed (directional) t-tests performed on the 
means. Thus,» r>cfb indicates that the mean for that variable in the random 
baseline was higher than the mean in the cfb condition, at p<0.05. n.s. 
indicates that differences in the means» if any, were not significant at 
p<0.05. 


all three projects. Subjects receiving only outcome feedback 
showed no Significant performance differences from the random 
baseline in any of the three projects. 

Table 10 summarizes staff productivity results. Tins 
information shows the actual productivity in tasks/man-day of 
the simulated staff as managed by the subject. Table ll 
contains the ANOVA results for the staff productivity data. 
Again, there was a significant difference in performance 
between subjects (F=106.88, p=0.0001) as well as within- 
subjects (F=109.59, p=0.0001). The Tukey Test for Additivity 
indicated no presence of interaction effects between the 
sequence of projects (p>0.9), nor the order of the projects 
(p>O0.7). Also, no interaction effects were present between 
the type of information provided and the type of project 
(p>0O.6), nor the type of information provided and the order of 


projects (p>0O.4). 
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TABLE 10. STAFF PRODUCTIVITY. Means and (Standard Deviations). 














N Ideal Undersize Fixedsize 
Cognitive Feedback 15 0.393 0.307 0.251 
(0.051 ) (0.047) (0.051 ) 
Feedforward 15 0.317 0.249 0.182 
(0.049) (0.051 ) (0.049) 
Outcome Feedback 15 0.249 0.177 0.112 
(0.051 ) (0.057) (0.037) 
Random Baseline 15 0.238 0.180 0.13] 
(0.060) (0.071) (0.032) 


TABLE 11. ANALYSIS OF VARIANCE. Dependent Variable: Staff Productivity. 








Source of Degrees of 

Variance S.o. freedom F-Value P R-Suare 
Model 1.62 57 8.89 0.0001 0.86 
Type of 

Information 0.43 2 106.88 0.0001 

Type of Task 0.44 2 109.59 0.0001 

Sequence 0.002 2 0.06 0.9443 

Order 0.006 2 0.09 0.7869 
Participant 0.71 41 1.37 0.0401 

Group*Task 0.005 G 0.66 0.6234 
Group*Order 0.007 4 0.98 0.4235 

Within Groups 0.16 77 





38 


IV. CONCLUSIONS 


A. SUMMARY OF RESULTS 

The objective of this study was to investigate the effects 
of cognitive feedback, outcome feedback, and feedforward on 
decision makers in a dynamic decision environment such as 
software project management. Chapter II (section B.2) pointed 
out that past research has shown the dysfunctional effects of 
outcome feedback. These dysfunctional effects have led 
researchers to seek alternative means to improve decision 
quality. Chapter II (section C) discusses two alternatives to 
outcome feedback: cognitive feedback and feedforward. Both 
cognitive feedback and feedforward seek to assist the decision 
maker in formulating a "mental model” of the task which 
confronts him/her. Without this model, decision makers have 
exhibited poor performance in handling the delays’ and 
oscillations associated with complex dynamic systems. 
Additionally, Chapter II (section C.3) explains why one would 
expect cognitive feedback to improve decision makers’ 
performance more than feedforward. The results of this study 
support these past findings. As the analysis of variance 
tests showed, there was a significant difference in subjects’ 
performance depending on the type of feedback with which they 


were provided. Subjects in the cognitive feedback condition 
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experienced less deviations from the initial project estimates 
than subjects in the feedforward or outcome feedback 
condition. Additionally, staffs managed by subjects receiving 
cognitive feedback showed greater productivity than staffs 
managed by subjects in the other feedback conditions. Also, 
tests comparing feedback groups with the random baseline 
showed that subjects in the cognitive feedback and feedforward 
conditions performed better than the random baseline in each 
of the three projects, whereas there was no. significant 
performance difference between subjects receiving outcome 


feedback and the random baseline. 


B. FEEDBACK AS A DECISION TOOL 
The results of thas experiment provide several 

implications for the design of project control systems, 
specifically those in support of software project managers. 
As Brehmer (1987) concluded from his DESSY experiment, 

These results show that system designers cannot rely upon 

the operators to develop good mental models ‘ff complex 

systems. This implies that we need to develop means that 

help the subjects develop such models, or possibly means 

that eliminate the need for predictive models. (p. 30) 
Brehmer provides several suggestions for improving the quality 
of information systems. The most important one, with respect 
to this study, states the need to communicate information 
about the system in nonverbal form. As this study showed, 


presenting information in graphical form did, in fact, raise 


the performance level of subjects receiving that information. 
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C. LIMITING FACTORS TO GENERALIZABILITY™ 

Although a majority of the subjects in this experiment had 
some managerial experience, the question remains whether they 
had enough experience to play the game. In other words, is it 
reasonable to make a comparison between their performance and 
that of real life software managers? 

Remus (1986) found, ina study to investigate the use of 
graduate students as surrogates for managers, that no 
Significant differences existed between the students) and 
managers in making production scheduling decisions. Although 
software project management is somewhat different from the 
task presented in Remus’ experiment, it is similar enough to 
apply his findings and assume that software engineering 
graduate students are reasonable surrogates sie this 
experimental investigation. 

The next limiting factor to consider is the nature of the 
particular project environment. One should take caution 
against generalizing the results presented in Chapter III to 
all types of project situations. In this experiment, each of 
the three projects was developed in a familiar in-house 
environment i.e., what is typically described as an organic- 


type project environment (Boehm, 1981, pp. 78-82). 


“This section is based on an unpublished paper by Abdel-Hamid, 
Sengupta, and Ronan (1990) which describes a similar experiment 
using the SDM gaming interface. 
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Finally, it is difficult to claim external valida tyne. 
laboratory-type studies. A review of the gaming literature by 
Remus (1978) does, however, indicate ccnsiderable similarity 
between decision making in games and managerial decision 
making per se. Since the project games in this experiment are 
a Simulation of three real life software projects, there 
seems to be no reason why there should be any exception to 


Remus’ general findings. 


D. FUTURE RESEARCH 

Based on the discussion in Section C above, one particular 
path for future research using the SDM gaming interface to 
investigate managerial decision making is evident. The above 
experiment could be replicated using real software project 
managers as the subjects. Although using graduate students as 
surrogates in research studies is useful, analyzing the 
behavior of experienced project managers could provide more 


Significant and ..oteworthy results. 
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APPENDIX 
EXPERIMENTAL INSTRUCTIONS PROVIDED TO SUBJECTS 
Al Written description given to subjects 
Introduction 


The exercise you are about to undertake is similar in many 
ways to flight simulators that pilots use to mimic flying an 
aircraft from takeoff at point A to landing at point B. 
Instead of flying the aircraft, though, this simulator mimics 
the life of a real software project from the start of the 
design phase until the end of testing. In this simulation, 
you will be more than an observer. In fact you will play a 
real role on the project: that of the project manager. 


Specifically, your role will be to track the project's 
progress using a number of reports that will be produced for 
you at different intervals during the project. You will then 
make the project's staffing decisions based on the knowledge 
you gain from these reports. As the project manager, you can 
hire additional staff or decrease the staffing level as you 
deem necessary to complete the project. Your objective (like 
that of any software project manager) is to manage your 
resources wisely and efficiently while always aiming to finish 
the project within budget and on schedule (plus any safety 
factor period available.) 


Projects 


You will be given three projects to manage, all of them real 
projects conducted in a real organization. The organization 
is on the leading edge in its software engineering practices. 
For each project, you will be given a project profile 
containing the following initial information: 


Estimated Project Size(in No. of tasks) 

Estimated Schedule Duration(in No. of Work Days) 
Estimated Project Cost(in No. of Man Days) 

Maximum Tolerable Project Duration(in No. of Work Days). 
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Your Task 


Your task is to use the reports generated by the project team 
at different points in the project to determine a desired 
staffing level for the remainder of the project. Your 
objective in setting the staffing level should be to decide on 
the best compromise between finishing on an acceptable 
schedule while avoiding an excessive cost overrun. 
Specifically, you should try to: 


(a) complete the project on schedule, 

(b) at the lowest possible cost, and 

(c) in any case, complete it before the maximum tolerable 
completion date. 


Your grade for the simulation will be based on an equal 
weighting of two factors: 


(a) The percentage by which the project overshoots' the 
original schedule. Thus, if the scheduled completion date 
for the project is 200 days, and your actual completion 
date is 240 days, you will be considered to have overshot 
the schedule by (240-200)/200 = 20%. 


(b) The percentage by which the project overshoots' the 
Original cost estimate. If the original cost estimate is 
2,000 man days and the actual cost of completion is 2,500 
man days, you will be considered to have overshot the cost 
by (2, 500=-2,000)7/2,000 = 257 


The following are some important things to consider in making 
your decisions: 


1. As the software project manager, you specify the desired 
staffing level. The actual staffing .evel may, of course, 
be different due to things you cannot control such as 
turnover and lengthy hiring delays. 


2. Each project is initialized with a particular core team of 
full time equivalent personnel (FTE). This is to reflect 
that fact that most projects start out with a small core 
team of personnel. For example, project 2 may be 
initialized to an FTE of 1.5. 


3. The hiring delay for new employees can take up to 6 weeks. 
Once new people are hired, the assimilation period fora 
newly hired employee is typically one month long. This is 
the time needed to train a new employee in the mechanics 
of the project and bring him/her up to speed. A new 
employee (i.e. one that is being trained) is only half as 
productive as an experienced employee. 


ee 


4. The personnel turnover rate is 20% per year. 


5. At different points in the project you will be given 
reported information on the status of the project. Two 
key pieces of information for this staffing task are: (1) 
The updated estimate of the total project cost in man days 
(this update can change to reflect the addition of new 
requirements and/or changes in the estimate of the team’s 
overall productivity); and (2) Effort expenditures to date 
(also in man days). Subtracting the second from the first 
yields the "Remaining Effort in man days." 


6. Let us say that at some point in the project the 
"Remaining Effort" 1s 1000 man days, the remaining time is 
100 days and you have 7 full time equivalent employees 
working. You are, thus, in a position where you have to 
use your judgement to do one of the following: 
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Stick with the current schedule. If so then you will 


need a staff size of 1000/100 = 10 full time 
employees. 

Stick with. your staff size of /7. This means the 
schedule has to be pushed back. In this case the 


model will make the appropriate adjustment to the 
schedule for you. That is extend it to 1000/7 = 143 
man days. 


Do a bit of both. That is increase the staff size a 
bit, say to 8, which will also mean that the schedule 
will be extended (appropriately by the model) to 
1000/8 = 125 days. 


How to Play the Game 


1. First, take some time to practice and get familiar with 
the system. 


(a) 
(b) 


(c) 


(d) 


Type DUMMY for running a dummy project. 
Run the dummy project for 1 interval. 


Go through all the options in the menu. Please be 
sure you understand all of them. 


When you are done, hit <ESC>. 
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The real simulation starts now. You will be given three 
projects, one at a time. When you are done with one 
project, you can move on to the next, till you have 
completed all three projects. 


For each project: 


(a) Insert the disk you are given, and enter a command 
from the A> prompt. The command for project 1 is 
PROJECT1, and so on. 


(b) The system will show you the size of the initial core 
team of senior designers (the full time equivalent 
number). It will then ask you for your initial 
desired staffing level. 


(c) Next it will run through the first simulation time 
period and show you the current reported statistics. 
Make your change to the full time equivalent staffing 
level on the documentation sheet provided after 
viewing the report. 


(d) Perform step (c) for as many intervals as necessary, 
till the project is complete. A project is considered 
complete when there are less than 40 days left for the 
project to be completed. That is, 

(New Estimate of Project Duration - Elapsed Time) < 40 
days. 

Thus, if the New Estimate of Duration = 426 days, and 
the Elapsed Time = 400 days, the project is considered 
complete. 


(e) There is no need to turn in the documentation sheet 
after each interval of a project. He aver, A LAB 
ATTENDANT MUST VERIFY YOUR FINAL RE: iTS at the 
completion of each project. Call a lab attendant as 
soon as you are done with a project. 


(£) Complete the appropriate questionnaire in your 
instruction booklet. 


(g) Move on to the next project. 


Rules of the Game 


You will be required to provide the new desired staffing 
level for the project at the beginning of every two-month 
interval (consisting of 40 work days). The simulation 
will stop to show current reported statistics and accept 
a desired staffing level after each 40 day work period. 
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Annotate your desired staffing level on the documentation 
sheet and then enter it at the simulation prompt. 


i YOU MUST WORK ALONE. You are not allowed to discuss this 
exercise with anyone other than a lab attendant. Also, 
please refrain from discussing this with any member in the 
other class until they have completed the exercise. 


ul Please follow the guidelines) strictly. The system 
prompts, along with instructions in this booklet, will 
guide you at every stage. 


* If you are in doubt about what to do next, ask for a lab 
attendant. 


Al.l Further instructions provided to cognitive feedback 
subjects 


How to use and Interpret the Plots 


Throughout the life of each project (starting with time 
elapsed = 40 days), you will have access to a series of plots 


providing information on the project. As the project 
progresses over time, the plots will be increasingly more 
meaningful in making your staffing decision. This and the 


next page explain how to interpret and use the plots in making 
your decisions. 


Plot on Project Size 
(refer to Chapter III, Figure 4) 


The figure below provides information on whether any schedule 
or budget slippage is a result of: (1) unexpected increases in 
the project's size (e.g., due to changes in users’ 
requirements), or (2) that we initially underestimated the 
effort required to complete the project (e.g. because we 
underestimated its complexity). In the first case, the 
Perceived Total Project Size (PJBSZ) will increase over time. 
In the second case, the Perceived Total Project Cost (JBSZMD) 
will increase over time. 
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Plot on Project Staffing vs Schedule 
(refer to Chapter III, Figure 5) 


The figure below provides feedback on the trade-off between 
minimizing schedule over-runs versus Minimizing cost 


over-runs. When a project runs into trouble, a manager can 
choose to stick with the project’s schedule (SCHCDT) by 
increasing the workforce level (FTEQWF). This always 
increases the cost of the project. On the other hand, a 


manager might desire to minimize his/her cost over-run, by 
avoiding an increase in the workforce level, and instead, 
increase the project's scheduled completion date. The 
indicated workforce level (WEFINDC) is an estimate of the 
workforce needed to stay on schedule. 


Plot on Workforce Mix 
(refer to Chapter III, Figure 6) 


A good feedback on why your costs may be higher than expected 
is to evaluate your staffing decision. In general, staffing 
decisions have the greatest impact on productivity. First, a 
larger workforce (FTEQWE) will, in most cases, be less 
productive because of the increases in communication and 
training overheads. Second, the workforce mix (1.e., percent 
of experienced vs new staff in the workforce) will also have 
an impact. The larger the percentage of experienced people in 
the workforce (FRWFEX), the more productive the workforce. 


Plot on Workforce Productivity 
(refer to Chapter III, Figure 7) 


This figure provides information on the average productivity 
of the team. In general, your staffing decisions will affect 
productivity in two ways. The larger the workforce you 
assemble (FTEQWF), the lower the average productivity 
(ASSPRD). This is because in a larger workforce people 
"waste" more time communicating with team mates. This 
communication overhead is plotted above. The communication 
overhead (COMMOH) curve tells you the percentage of a person's 
time that is wasted (on average) in communication with others. 


kk You are now ready to Start PROJECT 1 dts 
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A2 Information provided to subject for first project - order 
of projects varied depending on subject's group assignment 
(the order presented here is the same as the sequence 
given to subjects in Group 1: Ideal, Undersize, and 
then Fixedsize/bad estimate) 


PROJECT 1 
Management s Initial Project Estimates 
Initial Estimate of Project Size: 
Initial Estimate of Project Cost: 


Initial Estimate of Project Duration: 
Maximum Tolerable Project Duration: 


1,067 Tasks 
3,721 Man Days 
413 Days 

479 Days 


A project is considered complete when there are less than 40 
days left for the project to be completed. That is, (New 
Estimate of Project Duration - Elapsed Time) < 40 days. 
Staffing Level Sought (in FTE) 


Please enter your staffing decisions, i.e.,(In Full Time 
Equivalent) below: 


Initial Estimate: 


Time elapsed 40 days: 

Time elapsed 80 days: 

Time elapsed 120 days: 
Time elapsed 160 days: 
Time elapsed 200 days: 
Time elapsed 240 days: 
Time elapsed 280 days: 
Time elapsed 320 days: 
Time elapsed 360 days: 
Time elapsed 400 days: 
Time elapsed 440 days: 
Time elapsed 480 days: 
Time elapsed 520 days: 
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Time elapsed - 560 days: 
Time elapsed - 600 days: 


Time elapsed - 640 days: 


ae WHEN YOU ARE DONE, PLEASE CALL FOR A LAB ATTENDANT pple 


A3 Questions answered by all subjects after completion of 
first project 


1. Describe (in words, numbers, equation, etc) what decision 
rule you followed in deciding on the staffing level in 
this project: 


2. Please try to elaborate on the thinking process you went 
through in making your decisions in this project (use back 
of page if necessary): 


3. How clear were the instructions regarding the task? 
al Z 3 a 5 6 vi 8 9 
Not at all Very 
Clear Clear 


4. To what extent was the report on the progress of the 
project helpful in improving your own decision? 


1 2 3 4, 5 6 i 8 9 
Not at all Very 
Helpful Helpful 
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A3.1 Additional question asked of subjects in feedforward 
condition only 


5. To what extent was the training provided before the 
experiment helpful in improving your own decision? 


i Z 3 3 5 6 ty 8 9 
Not at all Very 
Helpful Helpful 


A3.2 Additional question asked of subjects in cognitive 
feedback condition only 


6. To what extent was the graphical information provided on 
the progress of the project helpful in improving your own 
decision? 

1 2 3 4 5 6 7 8 9 
Not at all Very 
Helpful Helpful 


“Ax “ PLEASE MOVE .ON- 10) PROJECT -275 *** 
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AZ Information provided to subject for second project - order 
of projects varied depending on subject's group assignment 


Project 2 


Management s Initial Project Estimates 


Initial Estimate of Project Size: 397 Tasks 
Initial Estimate of Project Cost: 1,460 Man Days 
Initial Estimate of Project Duration: 380 Days 
Maximum Tolerable Project Duration: 441 Days 


A project is considered complete when there are less than 40 
days left for the project to be completed. That is, (New 
Estimate of Project Duration - Elapsed Time) < 40 days. 


Staffing Level Sought (in ETE) 


Please enter your staffing decisions, i.e.,(In Full Time 


Equivalent) below: 


Initial Estimate: 


Time elapsed 40 days: 
Time elapsed 80 days: 
Time elapsed 120 days: 
Time elapsed 160 days: 
Time elapsed 200 days: 
Time elapsed 240 days: 
Time elapsed 280 days: 
Time elapsed 320 days: 
Time elapsed 360 days: 
Time elapsed 400 days: 
Time elapsed 440 days: 
Time elapsed 480 days: 
Time elapsed 520 days: 
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Time elapsed - 560 days: 
Time elapsed - 600 days: 
Time elapsed - 640 days: 
mo |) 6WHEN YOU ARE DONE, PLEASE CALL FOR A LAB ATTENDANT *** 
A5 Questions answered by subject after completion of second 


project were the same as those answered after the first 
project 


cD Bore MOV mr ON TOCRROJRC 1a ot a 
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A6 Information provided to subject for third project - order 
of projects varied depending on subject's group assignment 


Project 3 
Management s Initial Project Estimates 
Initial Estimate of Project Size: 
Initial Estimate of Project Cost: 


Initial Estimate of Project Duration: 
Maximum Tolerable Project Duration: 


1,866 Tasks 
2,972 Man Days 
362 Days 

420 Days 


A project is considered complete when there are less than 40 
days left for the project to be completed. That is, (New 
Estimate of Project Duration - Elapsed Time) < 40 days. 
Staffing Level Sought (in FTE) 


Please enter your staffing decisions, i.e.,(In Full Time 


Equivalent) below: 


Initial Estimate: 


Time elapsed 40 days: 
Time elapsed 80 days: 
Time elapsed 120 days: 
Time elapsed 160 days: 
Time elapsed z20O days: 
Time elapsed 240 days: 
Time elapsed 280 days: 
Time elapsed 320 days: 
Time elapsed 360 days: 
Time elapsed 400 days: 
Time elapsed 440 days: 
Time elapsed 480 days: 
Time elapsed 520 days: 
Time elapsed 560 days: 
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Time elapsed - 600 days: 
Time elapsed - 640 days: 
is WHEN YOU ARE DONE, PLEASE CALL FOR A LAB ATTENDANT el to 


A7 Questions answered by subject after completion of third 


project were the same as those answered after the first 
project 


A8 Questions answered by cognitive feedback subjects after 
completion of entire experiment (subjects in the 
feedforward and outcome feedback answered only questions 
et Cn tO, Aly ee, oo ana 14: 

1. In the projects that you just completed, did you 
(a) Use the project status report (Y/N)? 


(b) If you did, please describe how you used the 
information to make the staffing decision. 


Z. In the projects that you just completed, did you 
(a) Use the project staffing report (Y/N)? 


(b) If you did, please describe how you used the 
information to make the staffing decision. 
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4. 


In the projects that you just completed, did you 
(a) Use the plot on project size (Y/N)? 


(b) If you did, please describe how you used the 
information to make the staffing decision. 


In the projects that you just completed, did you 
(a) Use the plot on staffing and schedule (Y/N)? 


(6b) If you did, please describe how you used the 
information to make the staffing decision. 


In the projects that you just completed, did you 
(a) Use the plot on workforce mix (Y/N)? 


(6b) If you did, please describe how you used the 
information to make the staffing decision. 
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6. In the projects that you just completed, did you 
(a) Use the plot on workforce productivity (Y/N)? 


(b) If you did, please describe how you used the 
information to make the staffing decision. 


7. Have you in the past, participated in project management 


eNO 2 


8. If YES, to what extent was the task in this simulation 
Similar to your previous experience? 


iE Z 3 4 5 6 fi 8 2) 
Not at all Very 
Similar Similar 
9. How interesting was the task you just performed? 
1 2 3 4 5 6 7 8 S, 
Not at all Very 
Interesting Interesting 


10. How serious were you in performing the task? 


iL 2 3 4 5 6 vi 8 9 
Not at all Very 
Serious Serious 


ll. How clear were the instructions regarding the task, 
generally? 


al Z 6 4 5 6 7, 8 9 
Not at all Very 
Clear Clear 


12. How easy was the system to use? 


= Z 3 4 5 6 7 8 , 
Not at all Very 
Easy Easy 


oy 


13. Please give us some information about yourself (in 
absolute confidence. At no time will your name appear in the 
results. The data will only be used in an aggregate 
statistical sense). 

(a) Curriculum enrolled in: 

(b) Sex 

(c) Age 


(d) Full time work experience 
(in years) 


(e) How long ago (in years) did 
you complete your 
undergraduate education? 


(£) How familiar are you with computers, generally? 


i 2 3 4 S 6 / 8 9 
Not at all Very 
Familiar Familiar 


(g) How many hours (per week) do you use computers? 


14. Your general comments regarding the simulation: 


wits END OF SIMULATION cf 
Thank you for your participation. 
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